METHOD OF PROCESSING AN IMAGE SIGNAL WITH 
THE RESULT FROM DECISION ON 
A CORRELATION CORRECTED 



BACKGROUND OF THE INVENTION 
Field of the Invention 

[0001] The present invention relates to an image signal 
processing method advantageously applicable to a digital 
camera, image inputting unit or similar imaging apparatus 
of the type interpolating pixel data in virtual pixels by 
using the pixel data of real pixels. 

Description of the Background Art 

[0002] In a digital camera,, for example, the number of 
photosensitive cells or pixels bidimens ionally arranged 
on a solid-state image sensor determines the resolution 
of an image. Resolution increases with an increase in the 
number of pixels arranged in a limited area available with 
the image sensor. On the other hand, for a given number 
of photosensitive cells, the influence of noise , for example, 
decreases with an increase in the size and therefore 
sensitivity of the individual photosensitive cell, 
enhancing image quality. Stated another way, the size of 
the individual photosensitive cell decreases with an 
increase in the number of pixels that should be arranged 
in the limited area. As a result, sensitivity falls due 
to the decrease in the size of the individual photosensitive 
cell although the number of pixels and therefore resolution 
may increase . This limits image quality available with the 
image sensor. 



[0003] Japanese patent laid-open publication No. 
2000-184386, for example, discloses a signal processing 
method and a solid-state image sensor addressing the 
above-stated contradictory demands. Specifically, the 
image sensor disclosed in the above document includes 
photosensitive cells shifted from each other by one-half 
of a pixel pitch in the horizontal and vertical directions 
in a so-called honeycomb pattern. The honeycomb pattern 
allows the greatest possible number of pixels to be packed 
in a limited space. The individual photosensitive cell is 
provided with a relatively large size and therefore 
sufficient sensitivity for the number of pixels arranged, 
thereby protecting image quality from degradation . To use 
the resulting pixel data more effectively, pixel data are 
interpolated in virtual pixels where the photosensitive 
cells are absent, as distinguished from real pixels where 
the photosensitive cells are present. The real pixels and 
virtual pixels both are dealt with as pixels in order to 
further increase resolution . In addition , the frequencies 
of signal components of luminance data and color data are 
raised for thereby enhancing image quality. 

[0004] As for a virtual pixel, the prior art signal 
processing method described above detects a correlation 
between pixels in each of the oblique direction and vertical 
and horizontal directions and then selects one direction 
in which the degree of correlation is highest . Subsequently , 
the method interpolates pixel data in the virtual pixel 
by using pixels positioned in the direction having the 
highest degree of correlation, thereby improving image 
quality (adaptive interpolation) . 



[0005] However, the prior art interpolation of virtual 
pixels is apt to degrade image quality. For example, when 
a subject includes chromatic and achromatic lines over a 
particular width, a difference in level occurs in the 
direction of length due to such a configuration of the sub j ect . 
Also, when a resolution chart is shot, spurious signals 
intersecting the wedge-shaped portion of the chart, which 
is the index of resolution, appears in the resulting image. 
In addition, discontinuous lines appear in the image. 

SUMMARY OF THE INVENTION 

[0006] It is an obj ect of the present invention to provide 
a signal processing method capable of protecting image 
quality from degradation ascribable to the interpolation 
of virtual pixels. 

[0007] In accordance with the present invention , an image 
signal processing method uses a plurality of photosensitive 
cells each of which is shifted from the geometric center 
of the adjoining photosensitive cell by half a pitch in 
at least one of the direction of row and the direction of 
column for separating incident light representative of a 
scene into three primary colors and executing photoelectric 
transduction with separated light. The photosensitive 
cells are used as real pixels. Image signals output from 
the photosensitive cell are converted to digital image data . 
By using the digital image data, interpolation is executed 
for generating virtual pixels between the real pixels to 
thereby generate luminance data and chrominance data. When 
the image data are bidimensionally arranged such that green 
of the image data appears in a square lattice pattern, a 
first step obtains red pixel data from the real pixels facing 



Q 

5*: s 
. ;.' : s 



W3 



ru 



each other in one diagonal direction at both sides of green, 
and obtains blue pixel data from the real pixels facing 
each other in the other diagonal direction at both sides 
of green. When either one of a red real pixel and a blue 
real pixel is a subject pixel to be dealt with, a second 
step generates pixel data of a color complementary to green 
by using a plurality of pixel data surrounding the subject 
pixel and positioned in the direction of correlation, while 
making a decision for improving correlation accuracy. A 
third step generates pixel data to be interpolated in the 
virtual pixel from pixel data of the same color facing each 
other in at least one of the horizontal and vertical 
directions by using the pixel data of the complementary 
color and green pixel data . A fourth step limits a frequency 
band in the direction perpendicular to the direction used 
for the interpolation. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0008] The objects and features of the present invention 
will become more apparent from consideration of the 
following detailed description taken in conjunction with 
the accompanying drawings in which: 

FIG. 1 is a schematic block diagram showing a digital 
camera to which an image signal processing method embodying 
the present invention is applied; 

FIG. 2 is aview showing an image pickup section included 
in the illustrative embodiment, as seen from the light 
incidence side; 

FIG. 3 is a schematic block diagram showing a specific 
configuration of an interpolator included in the 
illustrative embodiment ; 

FIG. 4 is a schematic block diagram showing a specific 
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configuration of a band-broadening processor included in 
the illustrative embodiment; 

FIG. 5 is a flowchart useful for understanding a main 
routine unique to the illustrative embodiment; 

FIG. 6 shows a relation between the positions of real 
pixels and colors produced at the time of pickup included 
in the main routine; 

FIG. 7 is a flowchart showing a subroutine included 
in the main routine for executing interpolation signal 
processing for a still picture; 

FIG. 8 shows a flowchart useful for understanding a 
subroutine included in the subroutine of FIG. 7 for 
generating high-frequency luminance data; 

FIG. 9 shows a flowchart useful for understanding a 
subroutine included in the subroutine of FIG . 8 for executing 
adaptive processing; 

FIG. 10 is a flowchart showing a subroutine included 
in the subroutineof FIG. 9 for executing oblique correlation 
processing; 

FIG. 11A is a flowchart showing part of a 
vertical/horizontal correlation subroutine included in the 
subroutine of FIG. 9; 

FIG. 11B is a flowchart showing the other part of the 
vertical /horizontal correlation subroutine ; 

FIG. 12 shows a positional relation between magenta 
pixel data generated by the procedure of FIG. 8 and green 
pixel data G output at the time of a shot; 

FIG. 13 shows a pattern to occur when the green pixel 
data and magenta pixel data are dealt with as high-frequency 
luminance data ; 

FIG. 14 shows colors and signal levels derived from 
pixels at the time of a shot together with flags indicative 



of directions of high-frequency luminance data generated 
by a conventional algorithm; 

FIG. 15 shows portions of FIG. 14 corrected by 
vertical/horizontal correlation decision processing that 
is added to the procedure of FIG. 11; 

FIG. 16 shows a positional relation between pixels used 
for the comparison and correction of flag patterns shown 
in FIG. 8; 

FIG. 17 shows the result of comparison and correction; 

FIG. 18 shows a flowchart useful for understanding a 
subroutine included in the subroutine of FIG . 7 for executing 
luminance interpolation ; 

FIG. 19 shows a diagram useful for understanding the 
principle of LPF (Low-Pass Filter) processing applicable 
to the subroutine of FIG. 18; 

FIG. 20 shows an interpolated pattern produced by 
horizontal interpolation forming part of the LPF processing 
of FIG. 19; 

FIG. 21 shows an interpolated pattern produced by 
vertical interpolation forming the other part of the LPF 
processing of FIG. 19; 

FIG. 22 shows a flowchart useful for explaining 
procedures of a subroutine included in the subroutine of 
FIG. 7 for executing color interpolation; 

FIG. 23 shows a flowchart useful for understanding a 
subroutine included in the main routine of FIG. 5 for 
broadening frequency bands for a still picture; 

FIG. 24 shows a relation between a thin line included 
in a scene shot and spaced from adjoining thin lines by 
a relatively great distance, the colors and signal levels 
of pixel data, and flags; 

FIG. 25 shows portions of FIG. 24 corrected by the 



vertical /horizontal correlation decision processing added 
to FIG. 11; 

FIG. 26 shows a relation between thin lines included 
in a scene shot and spaced from each other by a relatively- 
small distance, the colors and signal levels of pixel data, 
and flags ; 

FIG. 27 shows portions of FIG. 26 corrected by the 
vertical/horizontal correction decision processing added 
to FIG. 11; 

FIG. 28 shows the result of comparison and correction 
of a flag pattern executed in FIG. 8; 

FIG. 29 shows apix el pattern paying attention to green; 

FIG. 30 shows a pixel pattern paying attention to 
magenta ; 

FIG. 31 shows a plain picture resulting from the 
interpolation of green; and 

FIG. 32 shows a plain picture resulting from the 
interpolation of magenta. 

DESCRIPTION OF THE PREFERRED EMBODIMENT 

[0009] Referring to FIG. 1 of the drawings, a digital 
camera to which the present invention is applied is shown. 
Part of the digital camera not directly relevant to the 
understanding of the present invention is not shown nor 
will be described. As shown, the digital camera, generally 
10, includes an optical lens system 12, an operation panel 
14, a system controller 18, and a timing signal generator 
20. The camera 10 further includes a driver 22, an image 
pickup section 26, preprocessor 28, a signal processor 30, 
a compander 32, a storage 34, and a monitor 36. 



[0010] The optical lens system 12 includes, e.g., a 
plurality of lenses, a zoom mechanism, and an AF (Automatic 
Focus) control mechanism although not shown specifically. 
The zoom mechanism adjusts the positions of the lenses and 
therefore a view angle in accordance with an operation s ignal 
14a fed from the operation panel 14. The AF control 
mechanism controls the distance between the camera 10 and 
a desired subject to thereby focus the former on the latter. 
The operation signal 14a is input to the system controller 
18 through a system bus 16 and a data bus 18a. The system 
controller 18 feeds control signals 18b to the system bus 
16. The control signals 18b are input to the timing signal 
generator 20 and driver 22 as control signals 18c, as will 
be described more specifically later. The timing signal 
generator 20 and driver 22 operate in accordance with the 
control signals 18c, feeding drive signals 22a to the lens 
system 12 . 

[0011] The operation panel 14 includes a shutter release 
button and a monitor although not shown specifically. The 
operator of the camera 10 can select desired one of various 
items appearing on the monitor by using a cursor. The 
operation panel 14 further includes a camera/movie selecting 
section that allows the operator to select either one of 
a camera or still picture mode and a movie picture mode. 
The camera/movie selecting section outputs a particular 
signal, not shown, representative of the camera mode or 
the movie mode. 

[0012] The system controller 18 includes, e.g., a CPU 
(Central Processing Unit) and a ROM (Read Only Memory) 
although not shown specifically. The ROM stores a program 
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describing the operation procedure of the camera 10 . The 
system controller 18 generates the control signals 18b in 
accordance with the above operation procedure and 
information fed from the operation panel 14 through a signal 
line 18a in response to operator's manipulation, i.e., an 
operation signal 14a . The control signals 18b are delivered 
not only to the lens system 12 but also to the timing signal 
generator 20 and driver 22 on the system bus 16 and signal 
line 18c. Further, the control signals 18b are fed to the 
preprocessor 2 8 , signal processor 30 , compander 32, storage 
34 and monitor 36 although not shown specifically. 

[0013] The timing signal generator 20 includes an 
oscillator, not shown, for generating a system clock on 
which the operation of the camera 10 is based. The 
oscillator may be implemented as a VCO (Voltage Controlled 
Oscillator) by way of example . The timing signal generator 
20 feeds the system clock to the system controller 18 and 
other blocks of the camera 10 that need the system clock. 
At the same time, the timing signal generator 20 divides 
the frequency of the system clock to thereby generate various 
signals necessary for the camera 10. 

[0014] More specifically, the timing signal generator 
20 includes a circuit for generating timing signals 20a 
and 20b in accordance with the control signal 18b. The 
timing signal 20a is fed to the driver 22 while the timing 
signal 20b is fed to the preprocessor 28 and signal processor 
30. The timing signal generator 20 additionally feeds 
various timing signals to the other blocks of the camera 
10 as well, although not shown in FIG. 1. 



[0015] The driver 22 generates drive signals 22a, 22b 
and 22c in accordance with the timing signal 20a . The drive 
signals 22a, 22b and 22c are fed to the lens system 12, 
mechanical shutter 24 and image pickup section 26, 
respectively. Particularly, the driver 22 generates the 
drive signal 22c in accordance with the camera mode or the 
movie mode selected by the operator. 

[0016] The mechanical shutter 24 is operated by the 
operator via the shutter release button provided on the 
operation panel 14. More specifically, when the operator 
presses the shutter release button, the operation panel 
14 feeds a control signal 14a to the system controller 18 
through the system bus 16. In response, the system 
controller 18 delivers the control signal 18b to the driver 
22 via the system bus 16 and signal line 18c. The control 
signal 18b causes the driver 22 to feed the drive signal 
22b to the mechanical shutter 24. 

[0017] The image pickup section 26 includes an optical 
low-pass filter 26a, a color filter 26b, and a solid-state 
image sensor 26c. The low-pass filter 26a and color filter 
26b are affixed to one side of the image sensor 26c to which 
light representative of a scene is incident. The image 
sensor 26c is implemented as a single, color CCD (Charge 
Coupled Device) image sensor by way of example . When a light 
image is focused on the image sensor 26c by the lens system 
12, the image sensor 26c outputs a signal 27 representative 
of the quantities of light incident to image sensing devices 
or photosensitive cells, which are arranged on the image 
sensor 26c. 
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[0018] More specifically, the low-pass filter 26a, color 
filter 26b and image sensor 26c are sequentially arranged 
in this order, as named from the light incidence side, and 
constructed integrally with each other. As shown in FIG. 
2, the color filter 26b, which is a single color filter, 
has color filter segments 260 corresponding one-to-one to 
image sensing devices or photosensitive cells 262 included 
in the image sensor 26c. The color filter segments 260 may 
be of three primary colors, i.e., red (R) , green (G) and 
blue (B) . FIG. 2 shows a specific arrangement in which the 
G filter segments 260 are arranged in a square lattice pattern 
while the filter segments 260 of the same color other than 
G diagonally face each other at opposite sides of each G 
filter segment 260. More specifically, the R filter 
segments 260 diagonally face each other at opposite sides 
of each G filter segment 260. This is also true with the 
B filter segments 260. The pattern shown in FIG. 2 will 
hereinafter be referred to as a honeycomb type G square 
lattice, RB full-checker pattern. 

[0019] In FIG. 2, six of fourteen pixels are assigned 
to G while each four of the same is assigned to each of 
R and B. It is to be noted that the term "square lattice" 
mentioned in relation to G does not refer to the shape of 
the individual pixel, but refers to the pattern in which 
theGpixelsare arranged . The individual pixel is polygonal , 
e.g., rectangular, hexagonal or octagonal. 

[0020] The image sensing devices may be implemented as 
MOS (Metal Oxide Semiconductor) type, solid-state image 
sensing devices in place of CCDs, as will be described 
specifically later. Signal charges are generated in the 



11 



image pickup section 26 by photoelectric transduction in 
accordance with the drive signal 22c and output at a 
preselected timing. For example, the signal charges are 
delivered to vertical transfer paths by field shift during 
vertical blanking period and then transferred to a 
horizontal transfer path by line shift. An output circuit, 
not shown, converts the signal charges output from the 
horizontal path to the signal 27 , which is an analog voltage 
signal 27. The signal 27 is input to the preprocessor 28. 
When the image sensor 26c is implemented by CCDs, it 
selectively reads all pixels or part of them by reduction. 

[0021] The preprocessor 28 includes a CDS (Correlated 
Double Sampling) circuit and an ADC ( Analog-to-Digital 
Converter) although not shown specifically. The CDS 
circuit reduces noise, i.e., 1/f noise and reset noise 
contained in the analog voltage signal 27 . The preprocessor 
28 may additionally execute gamma correction with the analog 
signal . The resulting analog signal free from noise is input 
to the ADC. The ADC converts the analog signal to a digital 
signal 29 by quantization using a preselected quantization 
level in accordance with the timing signal 20b, which is 
fed from the timing signal generator 20 . The timing signal 
20b may include a conversion clock. The digital signal 29 
is input to the signal processor 30. 

[0022] The signal processor 30 includes a data corrector 
30a with a frame memory function, an interpolator 30b, and 
a band-broadening processor 30c. The data corrector 30 
includes a frame memory or buffer memory and a correcting 
circuit although not shown specifically. The interpolator 
30b is constructed to execute gamma correction and white 
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balance adjustment. Gamma correction allows power 
consumption and circuit scale to be reduced for thereby 
reducing the number of bits to be dealt with. This gamma 
correction will be omitted if it is assigned to the 
preprocessor 28. 

[0023] The digital signal or image data 29 output from 
the preprocessor 28 are written to the frame memory of the 
data corrector 30a. The pixel data are repeatedly read out 
of the area of the frame memory that is sequentially shifted . 
In this respect, the frame memory should preferably be 
implemented as a nondestructive memory. The control 
signals 18b output from the system controller 18 include 
a control signal 18d input to the frame memory via the system 
bus 16. The control signal 18d includes some different 
signals for control, i.e., a write/read enable signal, a 
clock signal and so forth. The data corrector 30a adjusts, 
e.g., the gain of the individual filter segment in order 
to match the level of segment-by-segment image data to signal 
processing to follow. The image data stored in the frame 
memory are fed to the interpolator 30b in a preselected 
sequence as image data 38. 

[0024] In the illustrative embodiment, the interpolator 
30b is capable of interpolating both of a still picture 
and a movie. The interpolator 30b switches the destination 
of the image data 38 in accordance with the operator's 
intention. Reference will be made to FIG. 3 for describing 
specific circuitry included in the interpolator 30b for 
interpolating a still picture. 

[0025] As shown in FIG. 3, the circuitry is generally 
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made up of a luminance interpolating circuit 40 and a color 
interpolating circuit 42. The interpolator 30b 

interpolates high-frequency luminance data Y H containing 
high frequency components and primary color data, i.e., 
R, G and B data in real pixels and virtual pixels intervening 
between the real pixels. The image data 38 output from the 
data corrector 30a are input to the luminance interpolating 
circuit 40 and color interpolating circuit 42. It is to 
be noted that the image data 38 are derived from the real 
pixels of the image sensor 26c where the photosensitive 
cells exist. 



[0026] Specifically, the luminance interpolator 40 
includes a high-frequency luminance data generator 400 and 
a luminance data interpolator 402 . The high-frequency 
luminance data generator 400 produces high-frequency 
luminance data Y H (404) , which contain high frequency 
components, from the image data 38 at the positions of the 
real pixels or those of the virtual pixels. In the 
illustrative embodiment, the generator 400 calculates 
high-frequency luminance data Y H at the positions of the 
real pixels. More specifically, the generator 400 
determines a direction in which the pixel data 38 are 
correlated and then generates high-frequency luminance data 
Y H (404) in accordance with the direction of a correlation, 
as will be described in detail later. The luminance data 
interpolator 402 generates high-frequency luminance data 
Y H (44) at the positions of the virtual pixels by using the 
high-frequency luminance data Y H (404) . In this manner, 
the luminance interpolating circuit 40 interpolates the 
high-frequency luminance data Y H in all of the real pixels 
and virtual pixels and delivers the interpolated 
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high-frequency luminance data Y H (44) to the band-broadening 
processor 30c, FIG. 1. 

[0027] If desired, the luminance interpolating circuit 
40 may be constructed to generate high-frequency luminance 
data Y H for the virtual pixels and then generate 
high-frequency luminance data Y H for the real pixels by using 
them . 

[0028] The color interpolating circuit 42 executes, by 
taking account of the color arrangement of the real pixels, 
interpolation for colors to which neither the virtual pixels 
nor the real pixels correspond, thereby generating R, G 
and B color data over the entire picture. For this purpose, 
the color interpolating circuit 42 includes an R 
interpolator 420 , a G interpolator 422 , and a B interpolator 
424. R data 38a, G data 38b and B data 38c constituting 
the image data 38 are input to the R interpolator 420, G 
interpolator 42 2 and B interpolator 42 4, respectively . The , 
R, G and B interpolators 420 through 424 execute 
interpolation by using the respective color data 38a through 
38c to thereby generate R, G and B color data at the positions 
of all of the real and virtual pixels. Such interpolation 
will be described more specifically later. The R, G and 
B interpolators 420 , 422 and 424 respectively feed the 
resulting R, G and B color data 46 , 48 and 50 to the 
band-broadening processor 30c. 

[0029] As shown in FIG. 4 specifically, the 
band-broadening processor 30c includes a high-pass filter 
(HPF ) 52, an addition circuit 54, a color matrix 56, an 
anti-distortion circuit 58, an aperture adjuster 60, and 
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chrominance gain adjusters 62 and 64. The HPF 52 separates 
high frequency components Y h from the high-frequency 
luminance data 44. The high frequency components Y h are 
input to the addition circuit 54 as a signal 52a. As for 
the color G, a switch, not shown, may be connected between 
the HPF 52 and the addition circuit 54 so as not to directly 
feed the high frequency components to the color G, in which 
case the system controller 18 will feed a switching signal 
to the switch. 

[0030] The addition circuit 54 includes an R adder 540 , 
a G adder 542 and a B adder 544. The R, G and B adders 540, 
542 and 544 respectively receive the R, G and B color data 
46,48 and 5 0 via one end thereof 540a, 542a and 5 4 4a. Also , 
the R, G and B adders 540 through 544 receive the high 
frequency components Y h via the other ends thereof 540b, 
542b and 544c, respectively. With this configuration, the 
addition circuit 54 adds the high frequency components to 
each of the R, G and B color data to thereby broaden the 
frequency bands of the color data. In this connection, the 
G pixel data themselves may be considered to have been 
broadened in frequency band because they contain high 
frequency components. The addition circuit 54 feeds R, G 
and B color data 54a, 54b and 54c with the broadened frequency 
bands to the color matrix 56. 

[0031] The color matrix 56 produces luminance data Y and 
chrominance data or color data C r and C b from the R, G and 
B color data 5 4a , 5 4b and 54c by using conventional equations . 
The color matrix 56 feeds the luminance data Y (56a) and 
chrominance data C r (56b) and C b (56c) to the anti-distortion 
circuit 58 . 
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[0032] The anti-distortion circuit 58 includes three 
LPFs (Low-Pass Filters) 580, 582 and 584 to which the 
luminance data Y (56a) and chrominance data C r (56b) and 
Cb (56c) , respectively, are input. The LPFs 580 through 
584 are responsive even to high frequencies in order to 
obviate aliasing distortion without disturbing the 
frequency bands of the three input signals 56a through 56c. 
The LPF 580, among the others, passes up to the highest 
frequency in accordance with the luminance data Y . Further , 
the circuit 58 is configured to limit either one of a 
horizontal and a vertical frequency band overlapping each 
other, thereby protecting image quality from deterioration 
ascribable to overlapping frequencies. The circuit 58 
feeds luminance data (58a) and chrominance data C r (58b) 
and Cb (58c) sub j ected to the above processing to the aperture 
adjuster 60 and chrominance gain adjusters 62 and 64, 
respectively . 

[0033] The aperture adjuster 60 adjusts an aperture by 
LPF processing in such a manner as to prevent the high 
frequency components from falling. This processing is 
equivalent in effect to edge enhancement. The chrominance 
gain adj usters 62 and 64 regulate the gains of the chrominance 
data C r (58b) and Cb (58c) to a preselected level . The signal 
processor 30 feed luminance data Y (66a) and chrominance 
data C r (66b) and C b (66c) generated by the above processing 
to the compander 32. 

[0034] Referring again to FIG. 1, the compander 32 
includes a frame memory, a compression circuit and an 
expansion circuit although not shown specifically. The 
frame memory temporarily stores one frame of luminance data 
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Y and chrominance data C r and C b input thereto. The 
compression circuit compresses the picture in accordance 
with, e.g., JPEG (Joint Photographic Experts Group) 
standards. The expansion circuit expands the compressed 
picture to thereby restore the original image data. The 
frame memory may be implemented by the frame memory of the 
signal processor 30, if desired. A plurality of frame 
memories may advantageously be used in order to deal with 
a movie, i.e., frames continuously picked up. The JPEG 
standards may be replaced with MPEG (Moving Picture Experts 
Group) standards or motion JPEG standards that apply the 
JPEG standards to the individual frame of a movie. 

[0035] In a record mode, the compander 32 delivers the 
compressed data to the storage 34 over the system bus 16 
under the control of the system controller 18. Also, the 
compander 32 is capable of simply passing the luminance 
data Y (66a) and chrominance data C r (66b) and C b (66c) 
therethrough to the system bus 16 under the control of the 
system controller 18 . Further, the compander 32 is capable 
of transferring the signals output from the signal processor 
30 to the monitor 36 over the system bus under the control 
of the system controller 18 . In a play mode, the compander 
32 reads the compressed data out of the storage 34 over 
the system bus 16 and expands them. The expanded data are 
written to the frame memory and then read out in a preselected 
sequence under the control of the system bus 18. The data 
read out of the frame memory are delivered to the monitor 
36 and displayed thereby. 

[0036] The storage 34 includes a recording section for 
recording the image data in a recording medium and a 
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reproducing section for reading the image data out of the 
recording medium, although not shown specifically. The 
recording medium may be implemented by any one of Smart 
Media (trade name) or similar semiconductor memory, a 
magnetic disk, an optical disk and so forth. If use is made 
of a magnetic disk or an optical disk, then the storage 
34 will additionally include a modulating section for 
modulating the image data and a head for writing the image 
data . 

[0037] The monitor 36 displays the luminance data Y and 
chrominance data C r and C b or the R, G and B color data fed 
thereto over the system bus 16 under the control of the 
system controller 18 . At this instant, the monitor 36 takes 
account of its screen size and adjusts the display timing. 

[0038] As stated above, the camera 10 executes signal 
processing with a picture picked up and then records the 
processed picture . Also, the camera 10 reproduces recorded 
image data to thereby display them on the monitor 36 in 
the form of a picture, as desired. 

[0039] The operation of the camera 10 will be described 
hereinafter. FIG. 5 shows a specific main routine to be 
executed by the camera 10 and also concentrating on a still 
picture. First, the user of the camera 10 switches on the 
camera 10 and then selects desired one of a plurality of 
modes available with the camera 10. When the operator 
selects the cameramode, the camera 10 continuously displays 
a scene being picked up through the lens system 12 on the 
monitor 3 6 while executing digital processing. This pickup 
will be referred to as preliminary pickup as distinguished 
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from actual pickup to follow (step S10) . Assume that the 
operator presses the shutter release button of the operation 
panel 14 to a half -stroke position during preliminary pickup . 
Then, the system controller 18 sets exposure conditions 
and adjusts the focal distance between the camera 10 and 
a subject to be picked up. The signal processor 30 may set 
parameters relating to the exposure conditions. 

[0040] Subsequently, the operator presses the shutter 
release button to a full-stroke position at a desired timing 
so as to actually pick up the scene (step S12) . In response, 
the operation panel 14 feeds an operation signal 14a to 
the system controller 18 . On receiving the operation signal 
14a, the system controller 18 delivers preselected control 
signals 18b to the timing signal generator 20 and driver 
22 over the system bus 16 and signal line 18c. The camera 
10 operates in accordance with timing signals and drive 
signals output from the timing signal generator 2 0 and driver 
2 2 , respectively . The image sensor 26c reads signal charges 
out of the pixels corresponding to the R, G and B color 
filter segments of the color filter 26b, which are arranged 
as described with reference to FIG. 2. The image sensor 
26c then transforms the signal charges to analog R, G and 
B signals 27. The analog signals 27 are input to the 
preprocessor 28 in accordance with the timing signal 20b. 

[0041] The preprocessor 28 cancels noise contained in 
the analog signals 27 and then converts analog signals 27 
to digital signals or image data 29 (step S14) . The image 
data 29 are input to the signal processor 30. 

[0042] The signal processor 30 sequentially executes 
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data interpolation for a still picture (subroutine SUB1) 
and band broadening (subroutine SUB2) with the image data 
29. FIG . 6 shows an exemplary relation between the image 
data 29 as to pixel positions. In FIG. 6, octagonal solid 
lines and octagonal phantom lines are representative of 
real pixels and virtual pixels, respectively. Letters R, 
G and B contained in the pixels indicate the colors of signals 
output from the pixels. Further, suffixes attached to R, 
G and B each indicate the position of the associated pixel 
in a matrix. 

[0043] Before the data interpolation for a still picture, 
the data corrector 30a executes various kinds of correction 
including gamma correction and white balance correction 
with the pixel data each corresponding to a particular pixel . 
If desired, the data corrector 3 0a may include a frame memory 
in order to temporarily store the image data, in which case 
the pixels will be read out one by one and then subjected 
to the above correction. The corrected image data 38 may 
be again written to the frame memory or delivered to the 
interpolator 30b, as desired. 

[0044] The interpolator 30b executes interpolation for 
a still picture on the basis of the image data 38 input 
thereto. This interpolation effectively increases 

resolution in relation to the honeycomb type, G square 
lattice, RB full-checker pattern, as will be described more 
specifically later. High-frequency luminance data Y H (44) 
and plain color data 46, 48 and 50 output from the 
interpolator 30b are fed to the band-broadening processor 
30c . 



21 



[0045] The band-broadening processor 30c broadens the 
frequency band of each of the high-frequency luminance data 
Y H (44) and plain color data 46, 48 and 50. As a result, 
luminance data Y (66a) and chrominance data (B - Y) (66b) 
and (R - Y) (66c) broadened in frequency band are output 
from the band-broadening processor 30c. 

[0046] Subsequently, the image data 66a, 66b and 66c are 
compressed (step S16) . For compression, use may be made 
of the previously mention JPEG standards or MPEG standards 
in the movie mode or the JPEG standards in the camera mode, 
although not shown or described specifically. The 
compressed image data are written to the recording medium 
mounted to the storage 34 (step S18) . 

[0047] After the step S16, whether or not the camera 10 
should end operating is determined (stepS20) . If the camera 
10 should continue the operation (NO, step S20) , then the 
procedure returns to the step S10. If the camera 10 should 
end the operation (YES, step S20), then the power supply 
to the camera 10 is shut off. 

[0048] The interpolation subroutine SUBl will be 
described in detail with reference to FIG. 7. As shown, 
the subroutine SUBl includes subroutines SUB3 , SUB4 and 
SUBS assigned to the high-luminance data generator 400, 
luminance data interpolator 402 and color interpolating 
circuit 42, respectively. In the illustrative embodiment, 
the high-frequency luminance data generator 400 generates 
high-frequency luminance data Y H at the positions of the 
real pixels in a checker pattern (subroutine SUB3) . 
Subsequently, the luminance data interpolator 402 
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interpolates luminance data in the virtual pixels by using 
the high-frequency luminance data Y B ( subroutine SUB4 ) . On 
the other hand, assume that the high-frequency luminance 
data generator 400 generates high-frequency luminance data 
Y H at the positions of the virtual pixels. Then, the 
luminance data interpolator 402 interpolates luminance data 
in the real pixels. 

[0049] The color interpolating circuit 42 interpolates 
the colors (subroutine SUB5) . As a result, R, G and B pixel 
data 46, 48 and 50 are generated over the entire picture 
including not only the real pixels but also the virtual 
pixels . 

[0050] Reference will be made to FIGS. 8 through 11 for 
describing the subroutine SUB3 more specifically. First, 
as shown in FIG. 8, whether or not to execute adaptive 
processing is determined (substep SS30) . The adaptive 
processing refers to producing high-frequency luminance 
data Y H from the pixel data by taking account of whether 
or not the pixel data are correlated. The camera 10 should 
preferably urge the operator to determine whether or not 
to execute the adaptive processing on the operation panel 
14. For example, the camera 10 may automatically execute 
the adaptive processing when the operator does not make 
such a decision . The adaptive processing will be described 
in more detail later. 

[0051] If the adaptive processing should be executed (YES , 
substep SS30) , then a subroutine SUB6 is executed. If the 
adaptive processing should not be executed (NO, substep 
SS30) , then the data of four surrounding pixels positioned 
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in a square lattice pattern are averaged to generate 
high-frequency luminance data Y H (substep SS32) . 

[0052] After the subroutine SUB6 or the substep SS32, 
i.e., after high-frequency luminance data Y H has been 
generated for a certain pixel, whether or not interpolation 
has completed for one frame of real pixels is determined 
(substep SS34) . If the answer of the substep SS34 is 
negative (NO) , the procedure returns to the substep SS30. 
If the answer of the substep SS34 is positive (YES) , 
comparison and correction are executed with the one frame 
of pixel data on a flag pattern basis (substep SS36) , as 
will be described specifically later. 

[0053] Hereinafter will be described the adaptive 
processing that calculates high-frequency luminance data 
Y H on the basis of a correlation in the oblique direction, 
horizontal direction and/or vertical direction in 
accordance with the operator's setting. As shown in FIG. 
9, whether or not a correlation should be detected in the 
oblique direction is determined (substep SS60) . For this 
decision, use may be made of a flag set in the camera 10 
beforehand. If the answer of the substep SS60 is YES, then 
high-frequency luminance data Y H is calculated by actual 
correlation decision processing (subroutine SUB7) . If the 
answer of the substep SS60 is NO, then high-frequency 
luminance data Y H is calculated on the basis of a correlation 
in the vertical and/or the horizontal direction (subroutine 
SUB8) . 

[0054] After the subroutine SUB7 , whether or not to 
execute correlation decision processing with pixel data 
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derived from the subj ect pixel is determined (substep SS62) . 
If the answer of the substep SS62 is YES, then the subroutine 
SUB8 is executed; if otherwise (NO, substep SS62) , then 
the subroutine SUB6 ends . 

[0055] The correlation decision processing in the 
oblique direction will be described with reference to FIG. 
10. As shown, data to be compared is calculated (substep 
SS70) . The data to be compared is used to determine a 
direction in which pixel data surrounding subject pixel 
data to be subjected to adaptive processing are correlated. 
For example, assume that a color B 4 6 shown in FIG. 6 is the 
subj ect of adaptive processing . Then , data AG to be compared 
is produced from pixel data G 35 , G55, G37 and G57 surrounding 
the subject color B 46 by an equation: 

AG = IG35 + G57 - (G37 + G55) I Eq. (1) 

Even when the pixel data is the color R, the data AG is 
produced from surrounding G pixel data . The data AG produced 
by the Eq. (1) is greater one of two values having inclination 
in the right and left directions . 

[0056] Subsequently, whether or not the pixel data 
obliquely facing each other at both sides of the subject 
pixel data are correlated, i.e., whether or not they have 
an oblique correlation is determined (substep SS70) . For 
this decision, use is made of a reference value Jl . If the 
data AG to be compared is greater than or equal to the 
reference value Jl , then the above pixel data are determined 
to have an oblique correlation (YES, substep SS72) . In this 
case, the four pixel data G35, G55, G37 and G57 surrounding 
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the subject pixel B 46 are averaged to thereby generate 
high-frequency luminance data Y H for the subject pixel B 46 
(substep SS74) . If the data AG is smaller than the reference 
value Jl (NO, substep SS72), then a flag assigned to the 
other correlation decision processing is set ( subs tep SS7 6 ) . 

[0057] Even the above procedure is likely to generate 
a spurious color. In the illustrative embodiment, as for 
pixel data R positioned near the boundary of a pixel where 
a spurious color is likely to appear, high-frequency 
luminance data Y H is produced by the above-described 
calculation in order to desirably reduce spurious colors 
at the boundary in the whole picture. Adaptive 
high-frequency luminance data can also be generated for 
pixel data B 2 4 by calculating data to be compared and then 
determining a correlation, although not described 
specifically. Thereafter, the subroutine SUB7 ends. 

[0058] The decisionona vertical /horizontal correlation 
will be described hereinafter with reference to FIGS. 11A 
and 11B. Again, assume that the vertical/horizontal 
correlation processing is applied to the pixel or color 
B 4 e shown in FIG. 6. The processing is executed over an 
area 68 containing eight real pixels R26, G35 , G37 , R44 , R48, 
G55, G57 and R 6 e surrounding the subject pixel B 46 . The 
illustrative embodiment additionally takes account of an 
area 70, which surrounds the area 68, for the purpose of 
further enhancing accurate decision. More specifically, 
among the pixels lying in the area 70, only four real pixels 
Bo6r B 4 2, B410 and B 8 6 of the same color as the subject pixel 
B 46 and positioned at the four corners of the area 70 are 
used for the decision. The real pixels B 0 6 and B 8 6 face each 
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other in the vertical direction while the real pixels B42 
and B410 face each other in the horizontal direction. 

[0059] Particular data to be compared is calculated for 
each of the areas 68 and 70 (substep SS800) . Data ARB H/ 
ARB V , ACB H , ACB v , AGB H and/or AGB V is the data to be compared 
relating to the area 68. Data ACBB H and ACBB V are the data 
to be compared for the area 70. In each data, the first 
letter A indicates arithmetic operation. The second 
letters R and G indicate the color to be used for arithmetic 
operation each. The second letter C indicates comparison 
with the subject pixel. The third letter B indicates the 
color to be used for the arithmetic operation or the color 
of the subj ect pixel . The fourth letter indicates the color 
of the subj ect pixel . Further , the suffixes H and V attached 
to the third letter or the fourth letter indicate the 
horizontal direction and vertical direction, respectively. 

[0060] For example , the data to be compared for the subj ect 
pixel B 46 are produced by: 
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[0061] The data to be compared may alternatively be 
produced before the decision on a vertical/horizontal 
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correlation. As for the decision on a vertical/horizontal 
correlation, reference values J2 , J3 and J4 are set 
beforehand for each of the vertical and horizontal sets. 
The reference values J2 through J4 are selected by 
experience . 

[0062] Subsequently, whether or not the pixel data facing 
each other in the vertical direction with the intermediary 
of the subject data B 4 e are correlated is determined {substep 
SS802) . Specifically, a correlation value (ARB H - ARB V ) 
is produced from the data ARB V in the vertical direction 
and the data ARB H in the horizontal direction. If the 
correlation value (ARB H - ARB V ) is smaller than the new 
reference value J2 (NO, step SS802) , then it is determined 
that the above pixel data are not correlated in the vertical 
direction. This is followed by a decision on a horizontal 
correlation (substep SS806) . If the correlation value 
(ARB H - ARB V ) is greater than or equal to the reference value 
J2 (YES, substep SS802), then it is determined that the 
data are correlated in the vertical direction, i.e. , close 
to each other. This is followed by the generation of 
high-frequency luminance data Y H (substep SS804) . 

[0063] In the substep SS804, high-frequency luminance 
data Y H is calculated by using the subject pixel B 46 and 
pixels R 2 6 and R 6 6- Basically, high-frequency luminance 
data Y H can be represented by either one of pixel data G 
and a color complementary thereto, e.g., magenta (Mg) . 
Magenta Mg can be calculated as a mean value of 

(0.5*R 4- 0.5*B) of the pixel data R and B, as well known 
in the art. In this case, high-frequency luminance data 
Y H 46 is produced by: 
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Y H 46 = B 46 /2 + (R 26 + R 6 6)/4 Eq. (3) 



[0064] A flag "v" indicative of a vertical correlation 

is attached to the high-frequency luminance data Y H . After 
the substep SS804, the procedure determines that 
high-frequency luminance data Y H has been calculated for 
the pixel B 4 6, and then returns. 

[0065] In the substep SS806, whether or not pixel data 
facing each other in the horizontal direction with the 
intermediary of the subject pixel data B 4 6 are correlated, 
i.e., whether or not they have a horizontal correlation 
is determined. More specifically, whether or not a 
correlation value (ARB V — ARB H ) is smaller than the reference 
value J2 is determined. If the correlation value 

(ARB V - ARB H ) is smaller than the reference value J2 , then 
it is determined that the above pixel data are not correlated 
in the horizontal direction (NO, substep SS806) . This is 
followed by a decision on a vertical direction (substep 
SS810) . If the correlation value (ARB V - ARB H ) is greater 
than or equal to the reference value J2 , then it is determined 
that the data are correlated in the horizontal direction 

(YES, substep SS806) . This is followed by the generation 
of high-frequency luminance data Y H (substep SS808) . 

[0066] In the substep SS808, high-frequency luminance 
data Y H is calculated by using the subject pixel B 4 6 and 
pixels R 44 and R 48 • The high-frequency luminance data is 
produced by: 

Y H46 = B 46 /2 + (R 44 + R 48 )/4 Eq. (4) 
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[0067] The flag "h" indicative of a horizontal 
correlation is attached to the high-frequency luminance 
data Y H . After the step SS808 , the procedure returns by 
determining that high-frequency luminance data Y H has been 
calculated for the pixel B 4 e - 

[0068] In the substep SS810, a decision on a vertical 
correlation is made by using the real pixels around the 
subject pixel B 46 whose high-frequency luminance data Y H 
should be calculated. When the decision is to be made by 
reducing the distance between the subject pixel B 46 and the 
surrounding pixels, use is made of the data ACB H and ACB V 
and reference value J3 . If a correlation value (ACB H - ACB V ) 
is smaller than the reference value J3 , then it is determined 
that the surrounding pixels are not correlated in the 
vertical direction (NO, substep 810) . This is followed by 
a decision on a horizontal correlation (substep SS814, FIG. 
11B) via a connector A. If the correlation value 

(ACB H - ACB V ) is greater than or equal to the reference value 
J3 , then it is determined that the above data are correlated 
in the vertical direction (YES , substep SS810) . This is 
followed by the calculation of high-frequency luminance 
data (step SS812) . 

[0069] As for the area 68, the decision on a vertical 
correlation may be made by using the pixels nearest to the 
subject pixel B 46 , i.e., the G pixels, in which case the 
data AGB H and AGB V to be compared are used. Specifically, 
a correlation value (AGB H - AGB V ) is compared with the 
reference value J3 in the same manner as described above. 
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[0070] In the substep SS812, high-frequency luminance 
data Yh is calculated by using the subject pixel B 46 , pixels 
R 26 and R 66 and Eq . (3) . The flag "v" indicative of a vertical 
correlation is attached to the high-frequency luminance 
data Y H . After the substep SS812 f the procedure returns 
via a connector B by determining that high-frequency 
luminance data Y H for the pixel B 46 has been calculated. 

[0071] In the substep SS814, a decision on a horizontal 
correlation is made by using the real pixels around the 
subject pixel B 46 . Again, when the decision is to be made 
by reducing the distance between the subject pixel B 46 and 
the surrounding pixels , the data ACB H and ACB V and reference 
value J3 are used. A correlation value is (ACB V - ACB H ) . 
If the correlation value (ACB V - ACB H ) is smaller than the 
reference value J3 , then it is determined that the 
surrounding pixels are not correlated in the horizontal 
direction (NO, substep SS814). This is followed by a 
decision on a vertical correlation (substep SS818) . If the 
correlation value (ACB V - ACB H ) is greater than or equal 
to the reference value J3 , then it is determined that the 
pixels are correlated in the horizontal direction (YES , 
substep SS814) . This is followed by the calculation of 
high-frequency luminance data Y H (substep SS816). 

[0072] The decision on a horizontal correlation may also 
be made with respect to a narrower area by using pixels 
nearest to the subject pixel B 46 within the area 68, i.e., 
G pixels, in which case the data AGB H and AGB V will be used 
for comparison. More specifically, a correlation value 
(AGB V - AGB H ) will be compared with the reference value J3 
in the same manner as described earlier. 
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[0073] In the substep SS816, high-frequency luminance 
data Y H for the subject pixel B 46 is calculated by using 
the subject pixel B 46 , pixels R 44 and R 48 and Eq . (4) . The 
flag "h" indicative of a horizontal correlation is attached 
to the luminance data Y H . Thereafter, the procedure returns 
by determining that high-frequency luminance data Y H for 
the pixel B 46 has been calculated. 

[0074] As stated above, a decision on a vertical 
correlation and a decision on a horizontal correlation each 
are effected in two steps for the accuracy purpose. Also, 
the two-step decision is effected in a direction in which 
the area to be dealt with is narrowed toward the subject 
pixel . However , when an elongate subj ect with a particular 
width is picked up, adequate high-frequency luminance data 
Y H cannot sometimes be obtained, resulting in low image 
quality. In light of this, the illustrative embodiment 
makes a further decision on a correlation with respect to 
the area 70 broader than the area 68. 

[0075] Specifically, in the substep SS818, a decision 
on a vertical correlation is made by using real pixels around 
the subject pixel B 46 and lying in the broader area 70. 
Specifically, the subject pixel B 46 and four pixels B 0 e, B 42 , 
B 4i o and B 86 surrounding it are used for the decision. The 
four pixels around the subject pixel B 46 are of the same 
color as the subject pixel B 46 . A correlation value 
(ACBBh - ACBBy) calculated by using the data ACBB H and ACBB V 
is compared with the reference value J4 . If the correlation 
value (ACBBr - ACBBy) is smaller than the reference value 
J4 , it is determined that a correlation does not exist in 
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the vertical direction (NO , substep SS818). This is 
followed by a decision on a horizontal correlation (substep 
SS822) . If the correlation value (ACBB H - ACBB V ) is greater 
than or equal to the reference value J4 (YES) , then it is 
determined that a correlation exists in the vertical 
direction (YES, substep SS818) . This is followed by the 
calculation of high-frequency luminance data Y H (substep 
SS820) . 

[0076] In the substep SS820, high-frequency luminance 
data Y H for the subject pixel B 46 is calculated by using 
pixels R 2 6 and R66 and Eq. (3) . The flag xx v" indicative of 
a vertical correlation is attached to the luminance data 
Y H . Thereafter, the procedure returns by determining that 
high-frequency luminance data Y H for the subject pixel B 4 6 
has been calculated. 

[0077] Subsequently, in the substep SS822, a decision 
on a horizontal correlation is made by again using real 
pixels around the subject pixel B 4 6 and lying in the broader 
area 7 0 . Specifically , the subj ect pixel B 4 6 and four pixels 
B 0 6r B 42 , B 4 io andB 8 6 surrounding it are used for the decision . 
A correlation value (ACBB V - ACBB H ) calculated by using the 
data ACBB V and ACBB H is compared with the reference value 
J4 . If the correlation value (ACBB V - ACBB H ) is smaller 
than the reference value J4 , then it is determined that 
a correlation does not exist in the horizontal direction 

(NO, substep SS822) . This is followed by the calculation 
of a mean value of four surrounding pixels (substep SS826) . 
If the correlation value (ACBB V - ACBB H ) is greater than 
or equal to the reference value J4 (YES) , then it is 
determined that a correlation exists in the horizontal 
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direction (YES, substep SS822). This is followed by the 
calculation of high-frequency luminance data Y H (substep 
SS824) . 

[0078] In the substep 824 , high-frequency luminance data 
Y H is calculated by using the subject pixel B 46 , pixels R 2 e 
and R 6 e and Eq. (4) . The flag "h" indicative of a horizontal 
correlation is attached to the luminance data Y H . 
Thereafter, the procedure returns by determining that 
high-frequency luminance data Y H for the subject pixel B 46 
has been calculated. 

[0079] In the substep SS826, a mean value of four pixels 
around the subject pixel B 46 is produced. Specifically, 
high-frequency luminance data Y H 46 for the subject pixel 
B 4 6 is calculated by using pixel data derived from all real 
pixels except color G pixels that lie in the area 68: 

Y H 46 = B 46 /2 + (R 26 + R44 + R410 + Ree) Eq. (5) 

[0080] A flag " + " indicative of calculation performed 
without any correlation is attached to the high-frequency 
luminance data Y H - The flag " + " means that the color of 
the subject pixel B 4 6 and another color are added for 
generating magenta Mg . More specifically, the flag "4-" is 
indicative of the addition of all pixels surrounding the 
subject pixel B 46 - Thereafter, the subroutine SUB8 ends 
by determining that high-frequency luminance data Y H for 
the subject pixel B 46 has been calculated. By the procedure 
described above , Mg pixel data are generated at the positions 
of the colors R and B in addition to the G real pixels, 
which are derived from the shot , as shown in FIG . 12 . Because 
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the G and Mg pixel data can be dealt with as high-frequency 
luminance data Y H , high-frequency luminance data Y H are 
generated at the positions of all real pixels shown in FIG. 
13 . 

[0081] Luminance sometimes appears on a subject in an 
elongate pattern having a particular width, as stated 
earlier. In such a case, a difference in luminance level 
occurs in the direction of length. For example, as shown 
in FIG. 14, a difference in luminance level occurs in a 
position where a yellow (ye) portion 72 is sandwiched between 
two white (w) portions; the color G is not shown. In each 
white portion, image data of, e.g., three primary colors 
R, G and B are produced with substantially the same level. 
The yellow portion 72 (b+, R+ , b+, R+ , b + ) is represented 
by, e.g. , R and G pixel data of substantially the same level 
and B pixel data of a lower level Jb . As the f lag " + " indicates , 
high-frequency luminance data for the yellow portion 72 
are produced from a mean value of four pixels around the 
subject pixel. Assuming that high-frequency luminance 
data for the pixels R+ and b+ are Y H (R+) and Y H (b + ), 
respectively, then the luminance data Y H (R+) and Y H (b + ) are 
expressed as : 

Y H (R+) = R/2 + (B + b-hB + b) = R/2 + (B + b) /4 

Eq. (6a) 

Y H (b+) = b/2 + (R + R + R + R)/8 = R/2 + R/2 

Eq. (6b) 

A difference )Y H in level between the high-frequency 
luminance data Y H (R+) and Y H (b+) is produced by: 
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)Y H = Y H (R+) - Y H (b+) = (B - b)/4 Eq. (7) 

The difference )Y H is causative of irregularity. 

[0082] To solve the above problem, the illustrative 
embodiment additionally executes vertical /horizontal 
correlation decision processing with respect to the area 
70, FIG. 6. This decision processing uses pixels of the 
same color including the subject pixel. Specifically, in 
FIG. 14, the pixel data designated by arrows 74 are used. 
The arrows 74 show that the substep SS818 and successive 
substeps shown in FIG. 11B have executed correlation 
decisions with B pixels, which are indicated by bold frames 
in FIG. 15, by replacing the flag " + " with the flag "h". 
In this manner, one frame of high-frequency luminance data 
Y H is calculated. Subsequently, the flag pattern of the 
one frame of high-frequency luminance data is compared and 
corrected (substep SS36, FIG. 8). 

[0083] For the comparison and correction, thirteen real 
pixels including the subj ect pixels are used. Specifically, 
the thirteen real pixels shown in FIG. 16 lie in a region 
76 and consist of pixels ul , u2 and u3 at the top row, pixels 
ml and m2 at the medium row, and pixels dl , d2 and d3 at 
the bottom row. Whether or not to correct the flag of each 
of the R and B positions is determined by using eight pixels 
surrounding the subject pixel to be compared and corrected. 

[0084] More specifically, if the flags of the pixels at 
the top and bottom rows are identical or if the flags of 
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the pixels at the right and left columns are identical, 
then the flag of the subject pixel is replaced with the 
flag of such surrounding pixels. For example, assume that 
the flags of the pixels ul, u2 and u3 and those of the pixels 
dl , d2 and d3 all are "h" indicative of a horizontal 
correlation, or that the flags of the pixels ul , ml and 
dl and those of the pixels u3 , m2 and d3 all are "h" . Then, 
the flag of the subject pixel b is replaced with the flag 
xx h". This will be understood even more specifically from 
a color B h shown in FIG. 17. High-frequency luminance data 
Y H based on a horizontal correlation is calculated at the 
same time as the correction of the flag. 

[0085] On the other hand, assume that the flags of the 
pixels ul, u2 and u3 and those of the pixels dl , d2 and 
d3 or the flags of the pixels ul , ml and dl and those of 
the flags u3 , m2 and d3 all are "v" indicative of a vertical 
correlation. Then, the flag of the subject pixel Jb is 
replaced with the f lag "v" . At the same time , high-frequency 
luminance data Y H based on a vertical correlation is 
calculated. This is successful to obviate the difference 

)Y H in luminance level. 

[0086] The subroutine SUB4 for the interpolation of 
luminance data will be described in detail with reference 
to FIG. 18. As shown, high-frequency luminance data Y H for 
virtual pixels are generated on the basis of the 
high-frequency luminance data Y H generated at the positions 
shown in FIG. 13 (substep SS40) . The interpolation is 
executed vertically and hori zon tally for each virtual pixel . 
FIG. 19 demonstrates LPF processing that may be used for 
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the vertical and horizontal interpolation. The LPF 
processing will be briefly described hereinafter. 

[0087] As shown in FIG. 19, part (a) , real pixels d ( - 3 ) , 
d ( -i) , d ( i) and d (3) indicated by solid lines and virtual pixels 
d n <-4>, d(- 2 ), dn(o>, d n(2 ) and d n(4 > indicated by phantom lines 
alternate with each other . The virtual pixels d n <-4) , d n <-2) , 
d„ ( o>, dn(2> and d„( 4 > may be dealt with as pixels storing no 
data in consideration of their correspondence to the real 
pixels. More specifically, (logical) ZEROs are filled in 
the virtual pixels beforehand. For example, assume that 
the virtual pixel d n <o) is to be interpolated in the horizontal 
direction, and that a digital filter has tap coefficients 
of k 0 , ki , k 2 , k 3 , k 4 , . . . , k n . Then , high-frequency luminance 
data Yh(oj is expressed as: 

Y H <o) = ko*d n (o) + ki*(d<i> + d ( -i)) + k 2 *(d n <-2> + d n <2>) 

+ k 3 * (d<- 3 ) + d(3>) + k 4 *(d n( -4) + d n (4)) 

+ . . . k„* (dn(-n) + d n (n) ) Ec I • < 8 ) 

As FIG. 19, part (a), indicates, the tap coefficients are 
doubled because of the ZERO data alternating with the real 
pixels. This relation applies to the other virtual pixels 
dn{-4), d n <-2>/ d n (2) andd n {4) as well. Consequently, as shown 
in FIG. 19, part (b) , high-frequency luminance data Y H (-4> , 
Y H (-2>, Y H( 2) and Y H < 4 ) are generated for the virtual pixels 
d n <-4), d n( -2), d n (2) and d n ( 4 ) , respectively. 

[0088] After the substep SS40 , whether or not 
high-frequency luminance data have been interpolated in 
virtual pixels over the entire frame is determined (substep 
SS42) . If the answer of the substep SS42 is NO, then the 
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procedure returns to the substep SS40. If the answer of 
the substep SS42 is YES, then the procedure returns by 
determining that the interpolation of high-frequency 
luminance data has completed. FIG. 20 shows the 
high-frequency luminance data Y H generated for the real 
pixels and virtual pixels and represented by the colors 
G and Mg in particular rows . The LPF processing is executed 
in the vertical direction as well. As a result, as shown 
in FIG. 21, the colors G and Mg each are generated in 
particular columns . 

[008 9] The LPF processing described above may be replaced 
with vertical and horizontal interpolation taking account 
of a correlation between a subject pixel and real pixels 
surrounding it. If desired, extra processing for canceling 
either one of a horizontal frequency band and a vertical 
frequency band that overlap each other may be executed with 
the high-frequency luminance data Y H . 

[0090] Reference will be made to FIG. 22 for describing 
the color interpolation (subroutine SUB5) in detail. The 
color interpolation is assigned to the color interpolating 
circuit 42, FIG. 3. The R, G and B data 38 read out of the 
data corrector 30a are input to the R interpolator 420, 
G interpolator 422 and B interpolator 424, respectively. 
Pixel data for real pixels and virtual pixels are generated 
color by color on the basis of the above pixel data. 

[0091] Specifically, as shown in FIG. 22, G pixel data 
are interpolated in a substep SS500 first. Because use is 
made of the honeycomb type G square lattice, RB full-checker 
pattern shown in FIG. 6, real pixels storing existing G 
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pixel data are represented by solid octagons . On the other 
hand, pixels not storing G pixel data, i.e. , virtual pixels 
and pixels storing pixel data other than G pixel data are 
represented by phantom octagons. Letthepixelsnot storing 
G pixel data be also referred to as virtual pixels. For 
the interpolation, each four existing pixel data are used. 

[0092] FIG. 20 shows the above relation more specifically. 
As shown, assume a row one half of which is occupied by 
real G pixels and the other half of which is occupied by 
virtual pixels G3.2 , G44, Gi 6 , G xs , Gn 0 and G 112 f and a row next 
to the above row and not including the color G at all . Then, 
to interpolate G pixel data in, e.g. , a pixel G22 , four pixel 
data G11, G13 , G31 and G33 adjoining the pixel G22 are used. 
Likewise, to interpolate G pixel data in a pixel G 2 4, four 
pixel data G13 , G15, G33 and G 35 are used. For example, G 
pixel data G 2 i to be interpolated in a subject virtual pixel 
is produced from the data of two real pixels belonging to 
the same column as the pixel G21: 

G21 = (G11 + G31) /2 Eq. (9) 

[0093] By applying the pixel data G i3 and G33 to the Eq . 

(9) , it is possible to interpolate G pixel data in the virtual 
pixel G23 . As for the virtual pixel G12 , the data of two 
real pixels Gn and G i3 belonging to the same row as the pixel 
G12 are used to interpolate G pixel data produced by: 

G12 = (Gn + Gi3> /2 Eq. (10) 

[0094] To interpolate G pixel data in the virtual pixel 
G32/ the pixel data G31 and G 33 are applied to the Eq. (10) . 
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As for a pixel data G22 located surrounded by the four pixel 
data G11, G i3 , G31 and G33 , G pixel data is interpolated by 
using the four pixel data: 



G 22 = (G11 + G13 + G31 + G33) /4 



Eq. (11) 



[0095] 



By applying the pixel data G13 , G15, G33 and G35 



surrounding the virtual pixel G 2 4 to the Eq. (11) , it is 
possible to interpolate G pixel data in the virtual pixel 
G 2 4 . Because the pixel data G 23 has already been calculated, 
the pixel data G i4 and pixel data G 3 4 and G25 can be calculated 
in the same manner as described above. Such interpolation 
using real pixels is repeated to generate a plain picture 
including only the G pixel data. The above relation does 
not hold at the outermost edges of the plain picture. In 
light of this, a boundary value may be set beforehand in 
order to enhance strict interpolation. Data at the 
peripheral portion of a plain picture may not be calculated 
at all because such data do not lie in a valid screen range, 
if desired. 

[0096] The substep SS500 is followed by a substep SS502 
for calculating R pixel data. As shown in FIG. 6, real R 
pixel data appear only every other column, i.e., R 0 o, R04, 
Ros, R012, R 22, R26, R210/ FUo, R44, R46 and so forth. In this 
case, interpolation is implemented by two pixel data 
obliquely adj oining a subj ect , virtual pixel . For example , 
the pixel data of pixels R 0 o and R22 are used to generate 
R pixel data to be interpolated in a virtual pixel Rn : 



R11 = (Roo + R22) /2 



Eq. (12) 
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[0097] Likewise, by applying pixel data R 40 and R22 to 
the Eq. (12) , it is possible to generate R pixel data to 
be interpolated in a virtual pixel R31. Further, pixel data 
R44 and R 22 are applied to the Eq . ( 12 ) for generating R pixel 
data to be interpolated in a virtual pixel R 33 . Even R pixel 
data for virtual pixels R 15 and R35 can be calculated by such 
oblique interpolation if consideration is given to the 
existing pixel data R 2 e as well. 

[0098] Subsequently, the four pixels calculated by the 
immediately preceding step are used to interpolate R pixel 
data in a pixel surrounded by the four pixels (substep SS504) . 
For example, as FIG . 6 indicates, the pixel data R i3 , R i5 , 
R33 and R35 are used to calculate R pixel data for a virtual 
pixel R24 * 

R24 = (R13 + Ris + R33 + R 35 )/4 Eq. (13) 

[0099] In the same manner, R pixel data Ro2r R20 and R42 
can be calculated by interpolation because the relation 
represented by the Eq. (13) holds between the surrounding 
pixels. Stated another way, all the pixel data to be used 
for interpolation obliquely adjoin a subject pixel. 

[0100] After the step SS504, interpolation is effected 
by using, among the pixel data interpolated by the above 
sequence of steps, four pixel data vertically and 
horizontally adjoining a subject pixel (substep SS506). 
For example, four pixel data R 02 , R11, R13 and R22 positioned 
at the left and right of and above and below the pixel data 
R12 are used to calculate R pixel data: 
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Rl2 = (^02 + Rll + Rl3 + R22)/4 



Eq. (14) 



[0101] To calculate pixel data R i4 , R 2 3, R25, R34 and so 
forth to which the above-described positional relation 
applies, corresponding surrounding pixel data R are 
substituted for the Eq . (14) . Virtual pixels at the 
peripheral portion of a picture remain non-interpolated. 
Such virtual pixels each may be interpolated by use of three 
pixels adj oiningit. For example , pixel data R 0 i is produced 
from three adjoining pixels R 0 o and R02 and Rn: 

R01 = (Roo + R02 + Rn)/3 Eq. (15) 

Pixel data R 03 , R05, Rio, R30, R41, R43 and R 45 each are 
interpolated in the same manner as the pixel data R 0 i. By 
the procedure described above, a plane picture relating 
to the pixel data R is completed. 

[0102] After the substep SS506, the interpolation of B 
pixel data is executed (substeps SS508, SS510 and SS512) . 
The interpolation of B pixel data also includes oblique 
interpolation, center interpolation using four 
interpolated data, and center interpolation using four 
vertically and horizontally adjoining pixels. The 
substeps SS508 through SS512 are similar to the substeps 
SS502 through SS506, as will be seen from the positional 
relation between the R pixel data and the B pixel data. 
Specifically, as the suffixes attached to the colors 
indicate, the B pixel data are arranged in a pattern in 
which the R pixel data are sequentially shifted in the 
horizontal direction (direction of row) by each two columns . 
To calculate B pixel data to be interpolated in virtual 
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pixels, "2" is added to the suffix representative of a 
position in a matrix and attached to each of the pixel with 
the suffix of 2 and successive pixels. By executing 
interpolation by taking account of such a relation, it is 
possible to complete a plain picture relating to the pixel 
data B. 

[0103] After the substep SS512, whether or not plain 
pictures of all colors have been completed is determined 

(substep SS514) . If the answer of the substep SS512 is NO, 
then the procedure returns to the substep SS500. This 
decision may be made color by color, if desired. If the 
answer of the substep SS512 is YES, then the subroutine 
SUBS ends . 

[0104] Reference will be made to FIG. 23 for describing 
the band broadening processing. As shown, HPF processing 
is executed with the high-frequency luminance data Y H (44) 
in order to pass frequencies higher than a preselected 
frequency (substep SS200) . As a result, high-frequency 
components Y h included in the high-frequency luminance data 
are separated as a signal. The high-frequency components 
Yh are input to the addition circuit 54, FIG. 4, made up 
of the R, G and B adders 54 0 , 5 42 and 544. More specifically, 
the R, G and B adders 540 , 542 and 544 respectively receive 
the plain, R pixel data (46) , G pixel data (48) and B pixel 
data (50) at one end 540a, 542a and 544a. Also, the adders 
540 through 544 receive the high-frequency components Y h 
corresponding to the above R, G and B pixel data at the 
other ends 540b, 542b and 544b. 

[0105] The addition circuit 54 adds the high frequency 
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components Y h and R, G and B pixel data pixel by pixel (substep 
SS202) . Generally, the R, G and B pixel data have their 
frequency bands broadened with the individual pixel being 
representedby amatrix position (i, j) . The band broadening 
operation will be represented by a suffix H hereinafter. 
Pixel data R H ij , G H ij and B H ±j with broadened frequency bands 
are expressed as : 



RHij 


= Rij 


+ 


Yhij 


Eq. 


(16a) 


GHij 


= Gij 


+ 


Yhij 


Eq. 


(16b) 


BHij 


- Bij 


+ 


Yhij 


Eq . 


(16c) 



[0106] The pixel data R H ij , G H ij and B H ij are input to the 
color matrix 56. The color matrix 56 executes luminance 
and chrominance of matrix calculation with the pixel data 
RHij , G H ij and B H ij at the positions of the real pixels and 
virtual pixels ( subs tep SS2 0 4 ) . Asaresult, luminance data 
Y and chrominance data C r and C b with broadened frequency 
bands are generated. 



[0107] The luminance data Y and chrominance data C r and 
Cb are fed to the anti-distortion circuit 58 . The 
anti-distortion circuit 58 executes anti-distortion 
processing with the luminance data Y and chrominance data 
C r and C b (substep SS206). Specifically, the circuit 58 
passes the luminance signal Y and chrominance signal C r and 
C b extended in frequency while executing LPF processing for 
obviating aliasing distortion. Further, the circuit 58 
limits either one of a horizontal and a vertical frequency 
band overlapping each other, thereby protecting image 
quality from degradation ascribable to overlapping 
frequencies . 
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[0108] The luminance data Y subjected to the 
anti-distortion processing is input to the aperture adj uster 
60 . The aperture adjuster 60 executes aperture adjustment, 
which corresponds to contour enhancement , with the luminance 
data Y (substep SS208) . On the other hand, the chrominance 
data C r and C b subjected to the anti-distortion processing 
are input to the chrominance gain adjusters 62 and 64, 
respectively. The chrominance gain adjusters 62 and 64 
respectively adjust the gains of the chrominance data C r 
and C b (substep SS210) . 

[0109] After the step SS210, whether or not one frame 
has been fully processed is determined (substep SS212). 
If the answer of the substep SS212 is NO, then the procedure 
returns to the substep SS200, if otherwise, the substep 
SUB2 ends. 

[0110] The procedure described above successfully 
enhances accurate correlation processing and thereby 
insures high image quality. For example, even when a scene 
including yellow thin lines sandwiched between white 
portions are picked up, steps that would degrade image 
quality do not appear in the thin lines. 

[0111] Thin lines with steps are apt to appear even in 

cases other than one stated above . For example , steps appear 
when the distance between thin lines lying in a dark portion 
sandwiched between light portions is relatively great or 
relatively small. As for steps to appear when the above 
distance is great, FIG. 24 shows flags obtained in up to 
the substep SS816 of the subroutine SUB8 . 



46 



[0112] in FIG. 24, capital letters indicate pixel data 
with a high signal level while small letters indicate pixel 
data with a low signal level. As shown, the colors R, G 
and B in light portions sandwiching a dark portion have 
a high signal level; R and B pixel data between the G pixel 
data each are the mean of four surrounding pixels. The R 
and B pixel data closest to the dark portion show that a 
horizontal correlation exists. In the dark portion, r and 
b pixels between g pixels each are the mean of four 
surrounding pixels. High-frequency luminance data Y H (r+) 
andY H (b+) cor responding to the r and b pixel s , respectively, 
are produced by: 

Y H (r+) = r/2 + (B + b + B + b)/8 

= r/2 + (B + b) /4 Eq. (17a) 

Y H (b+) = b/2 + (R + r + R + r)/8 

= b/2 + (R + r) /4 Eq. (17b) 

[0113] As for a difference in level between luminance 
data in the direction of length, a difference )Y H in level 

between the high-frequency luminance data Y H (R+) and Y H (b + ) 
is expressed as : 

)Y H = Y H (r+) - Y H (b+) = (R - R + B - b)/4 

Eq. (18) 

[0114] Assume that the substeps SS818 through SS824 are 
executed in consideration of the above situation. Then, 
the flag " + " attached to the pixel data derived from a mean 
of four surrounding pixels each and sandwiched between the 
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colors G and gis determined to have a horizontal correlation 
and therefore replaced with the flag "h". The pixel data 
to which the flag "h" is attached in place of the flag " + " 
are indicated by bold frames in FIG. 24. When the pixel 
data with the flag "h" are used to calculate high-frequency 
luminance data Y H , the difference in level represented by 
the Eq. (18) becomes zero. The flags attached to the pixel 
data R, B, r and b are related in the same manner as the 
pixel data G and g, so that the comparison or the correction 
of the flag patterns is not necessary. 

O [0115] As shown in FIG . 2 6 , in the case where the distance 

hi between thin lines is relatively small, the pixels r and 

5 LI 

O b between the light portions each are determined to be a 

2 mean of four surrounding pixels, as indicated by the flag 

^3 w + In this case, high-frequency luminance data Y H (r+) 

M and Y H< b + > are produced by the Eqs . (17a) and (17b), 

y> respectively, so that a difference in level )Y H is produced 

'% by the Eq. (18) . The substeps SS818 through 824 are so 

H executed as to obviate the above difference in level. 

However, as for the flags attached to the pixels r and b, 
such processing directed toward correlation accuracy cannot 
improve the situation shown in FIG. 26 at all (see FIG. 
27) . 

[0116] In light of the above, in the illustrative 
embodiment, the comparison and correction of flag patterns 
is executed after the adaptive processing (substep SS36) . 
Specifically, the pixel data B and R overlying and underlying 
the subject pixels are provided with the flag "h" each. 
Therefore, the flag attached to the subject pixels is 
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replaced with the flag "h" . By generating high-frequency 
luminance data Y H in accordance with the correction of the 
flag, it is possible to obviate a difference in level between 
the high-frequency luminance data Y H of the pixels indicated 
by bold frames in FIG . 28. In this manner , the illustrative 
embodiment generates high-frequency luminance data Y H 
without any difference in level and thereby insures high 
image quality. 

[0117] The illustrative embodiment generates 
high-frequency luminance data Y H over the entire frame by 
executing interpolation with the colors G and Mg in the 
horizontal and vertical directions, as stated above. 
Alternatively, the interpolation may be executed by paying 
attention to each of the colors G and Mg . FIGS. 29 and 30 
respectively show a pattern containing only the color G 
and a pattern containing only the color Mg . FIGS. 31 and 
32 respectively show plain, G pixel data and plain, Mg pixel 
data produced by the interpolation of the patterns shown 
in FIGS. 29 and 30. Further, as for the interpolation of 
luminance, high-frequency luminance data Y H may be 
implemented by data produced by adding corresponding plain , 
G and Mg pixels . 

[0118] As stated above, assuming, e.g., a chromatic or 
achromatic elongate subject, the illustrative embodiment 
detects a correlation between a subject pixel and pixels 
of the same color as the subject pixel over a broad range. 
The illustrative embodiment can therefore obviate a 
difference in luminance level in the direction of length. 
Further, the illustrative embodiment enhances correlation 
accuracy by considering the pattern of flags indicative 
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of the kind of a correlation each. This successfully 
obviates spurious signals intersecting the wedge shaped 
portion of a resolution chart and obviates discontinuous 
straight lines as well as other defects. 



[0119] In summary, in accordance with the present 
invention, green pixel data read out are regarded as 
luminance data and used as they are. Decision processing 
for enhancing correlation accuracy is executed with a 
sub j ect pixel. For example , whether or not a sub j ect pixel 
and pixels of the same color as the subject pixel are 
correlated is determined over a broad range; in this case, 
correction is executed that takes account of the pattern 
of flags indicative of the kind of a correlation each. A 
plurality of other pixel data around the subject pixel 
positioned in the direction of correlation are used to 
generate pixel data of a color complementary to green . More 
specifically, assume that one of a red and a blue real pixel 
is a subject pixel. Then, magenta pixel data, which is 
complementary to green, is generated by executing addition 
with the red pixel by taking account of blue surrounding 
it or executing addition with the blue pixel by taking account 
of red surrounding it. The magenta pixel data is also 
regarded as luminance data. This obviates, when a subject 
including chromatic or achromatic thin lines is picked up, 
a difference in level between the luminance data of pixels 
adj oining each other in the direction of length . Such pixel 
data is used to interpolate a virtual pixel by use of pixel 
data of the same color as the subject pixel and adjoining 
the sub j ect pixel in the vertical or the horizontal direction. 
At the same time, a frequency band in the direction 
perpendicular to the direction used for the above 
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interpolation is limited in order to protect image quality 
from degradation. The present invention can therefore 
obviate spurious signals intersecting the wedge-shaped 
portion of a resolution chart and can obviate discontinuous 
straight lines as well as other defects. 

[0120] The entire disclosure of Japanese patent 
application No. 2000-390121 filed on December 22, 2000, 
including the specif ication , claims, accompanying drawings 
and abstract of the disclosure is incorporated herein by 
reference in its entirety. 

[0121] While the present invention has been described 
with reference to the particular illustrative embodiment, 
it is not to be restricted by the embodiment. It is to be 
appreciated that those skilled in the art can change or 
modify the embodiment without departing from the scope and 
spirit of the present invention. 
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